Skip to content

WIP Hypershift BM - Agent based via Assisted Installer with IP echo server - #83029

Open
asood-rh wants to merge 7 commits into
openshift:mainfrom
asood-rh:hypershift-agent-bastion
Open

WIP Hypershift BM - Agent based via Assisted Installer with IP echo server#83029
asood-rh wants to merge 7 commits into
openshift:mainfrom
asood-rh:hypershift-agent-bastion

Conversation

@asood-rh

@asood-rh asood-rh commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Summary by CodeRabbit

  • Adds a twice-monthly bare-metal HyperShift agent test job for OVN-Kubernetes with Assisted Installer.
  • Provisions a Python HTTP IP echo service on the provisioning host and publishes its host IP and URL for test use.
  • Adds failure-tolerant cleanup that removes the service, systemd unit, and executable after testing.
  • Updates the LVM channel to stable-5.0 to match the available catalog index.
  • Adds ownership metadata for the new CI step registry components.

 Add metal-ds-hypershift-agent-mce-guest-f14 job for OCP 5.0

 Add HyperShift agent MCE guest debug job to the 5.0 amd64-nightly
 config with a 24h wait step for cluster inspection. Update
 MCE_VERSION to 2.12 and set LVM_OPERATOR_SUB_CHANNEL to auto-detect
The metal-ds-hypershift-agent-mce-guest-f14 job used MCE_VERSION 2.12,
which doesn't exist in the mce-dev-catalog. Replace it with
metal-ds-hypershift-agent-ovn-egressip-debug using MCE 2.17 (the
correct version for OCP 5.0), and add ipecho provision/deprovision
step-registry refs that deploy a Python HTTP server on the dev-scripts
provisioning host (192.168.111.1:9095). The ipecho server sits on the
same L2 segment (ostestbm bridge) as guest cluster worker VMs, enabling
EgressIP SNAT verification without routing or NAT in the path.
@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 6, 2026
@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

The change adds HyperShift agent OVN ipecho provision and deprovision steps. It adds a scheduled bare-metal OVN egress-IP debug job with provisioning, a wait step, and cleanup.

Changes

HyperShift OVN ipecho debug flow

Layer / File(s) Summary
Provision the ipecho service
ci-operator/step-registry/hypershift/agent/ovn/ipecho/provision/*
Adds a Python HTTP service, systemd configuration, readiness checks, shared outputs, step metadata, and ownership.
Run the scheduled OVN debug job
ci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-5.0__amd64-nightly.yaml
Adds a twice-monthly bare-metal HyperShift agent OVN egress-IP debug job with assisted provisioning, a wait step, deprovisioning, and a 20-hour timeout.
Deprovision the ipecho service
ci-operator/step-registry/hypershift/agent/ovn/ipecho/deprovision/*
Adds cleanup configuration and removes the service and files while absorbing cleanup errors.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ScheduledJob
  participant ProvisionStep
  participant DevScriptsHost
  participant IPEchoService
  participant WaitStep
  participant DeprovisionStep

  ScheduledJob->>ProvisionStep: Start ipecho provisioning
  ProvisionStep->>DevScriptsHost: Install and configure service
  DevScriptsHost->>IPEchoService: Start systemd service
  ProvisionStep->>IPEchoService: Check readiness
  IPEchoService-->>ProvisionStep: Return source IP
  ScheduledJob->>WaitStep: Run wait test
  ScheduledJob->>DeprovisionStep: Start cleanup
  DeprovisionStep->>DevScriptsHost: Stop and remove service
Loading

Possibly related PRs

Suggested labels: rehearsals-ack

Suggested reviewers: jparrill, memodi


Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error, 1 warning)

Check name Status Explanation Resolution
No-Sensitive-Data-In-Logs ❌ Error The provision script prints the private bridge URL/IP and curl response, while the ipecho service logs every client source IP and failure path dumps its journal. Do not print IP responses or internal URLs; redact client addresses and restrict failure diagnostics to sanitized service status without request logs.
Ipv6 And Disconnected Network Test Compatibility ⚠️ Warning The new EgressIP job sets IP_STACK=v4, and its ipecho helper hardcodes 192.168.111.1 and an IPv4-only HTTP URL. IPv6 and disconnected network compatibility notice: run /payload-job periodic-ci-openshift-release-master-nightly-4.22-e2e-metal-ipi-ovn-ipv6; use IP-family detection and IPv6-safe host-port formatting.
✅ Passed checks (13 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: adding an Assisted Installer HyperShift bare-metal test with an IP echo server.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The feature diff adds CI YAML, shell scripts, and step metadata only; searches found no Ginkgo Describe, Context, When, or It declarations or dynamic test titles.
Test Structure And Quality ✅ Passed The full branch diff contains only YAML, shell, JSON, and OWNERS files; it adds no Ginkgo test code, so the requested Ginkgo quality checks are not applicable.
Microshift Test Compatibility ✅ Passed The PR adds CI YAML, OWNERS, step metadata, and shell scripts only; the full PR diff adds no Go files or Ginkgo declarations, so this check is not applicable.
Single Node Openshift (Sno) Test Compatibility ✅ Passed The PR adds CI YAML, generated job YAML, shell scripts, metadata, and OWNERS files only; it adds no Ginkgo It/Describe/Context/When test.
Topology-Aware Scheduling Compatibility ✅ Passed The diff adds CI job/step configuration and host-side ipecho scripts only; no deployment/controller code or pod scheduling constraints such as affinity, spread, selectors, tolerations, replicas, or...
Ote Binary Stdout Contract ✅ Passed The PR changes only CI YAML, shell helpers, metadata, and OWNERS; it adds no OTE binary or suite code. The job uses the generic shell-based wait step.
No-Weak-Crypto ✅ Passed The added ipecho scripts and job contain no MD5, SHA-1, DES, 3DES, RC4, Blowfish, ECB, crypto implementation, or secret comparison.
Container-Privileges ✅ Passed Feature diff contains no privileged:true, hostPID, hostNetwork, hostIPC, SYS_ADMIN, or allowPrivilegeEscalation settings; root SSH is for documented systemd installation on the provisioning host.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@openshift-ci
openshift-ci Bot requested review from jparrill and memodi August 6, 2026 11:15
@openshift-ci

openshift-ci Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: asood-rh
Once this PR has been reviewed and has the lgtm label, please assign enxebre for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the do-not-merge/invalid-owners-file Indicates that a PR should not merge because it has an invalid OWNERS file in it. label Aug 6, 2026
@asood-rh

asood-rh commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-metal-ds-hypershift-agent-ovn-egressip-debug

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@asood-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

  - Reorder steps sections alphabetically (post/pre/test) to satisfy
    ci-operator-config-metadata validation
  - Add OWNERS files to intermediate directories hypershift/agent/ovn/
    and hypershift/agent/ovn/ipecho/ for step-registry-metadata check
  - Set LVM_OPERATOR_SUB_CHANNEL to stable-4.22 since stable-5.0
    channel does not exist yet
@asood-rh

asood-rh commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-metal-ds-hypershift-agent-ovn-egressip-debug

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@asood-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

  - Strip trailing newlines from metadata.json files to match
    make registry-metadata output format
  - Reorder steps sections alphabetically (post/pre/test) for
    ci-operator-config-metadata validation
  - Add OWNERS files to intermediate directories hypershift/agent/ovn/
    and hypershift/agent/ovn/ipecho/
  - Change LVM_OPERATOR_SUB_CHANNEL from stable-4.22 to stable-4.21
    to match other working OCP 5.0 jobs
@asood-rh

asood-rh commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-metal-ds-hypershift-agent-ovn-egressip-debug

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@asood-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

The operatorhub-subscribe-lvm-operator step defaults to subscribing
from the redhat-operators catalog, which lacks the required channel
for OCP 5.0. The lvms-catalogsource step already creates a custom
catalog source using the correct index image, but the subscription
was not pointed at it. Add LVM_OPERATOR_SUB_SOURCE: lvm-catalogsource
to match all other working OCP 5.0 jobs
@asood-rh

asood-rh commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-metal-ds-hypershift-agent-ovn-egressip-debug

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@asood-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

The lvm-catalogsource index image (v5.0) only contains channels
stable-5.0 and stable-4.22. The subscription was failing because
stable-4.21 does not exist in that index. Other OCP 5.0 jobs use
stable-4.21 with the redhat-operators catalog, but our job uses
lvm-catalogsource which has different channel availability.
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@asood-rh, pj-rehearse: unable to determine affected jobs. This could be due to a branch that needs to be rebased. ERROR:

couldn't prepare candidate: couldn't checkout base SHA bbd3c9790981144a4e74ef5403ee71d77a852904: error checking out "bbd3c9790981144a4e74ef5403ee71d77a852904": exit status 128 fatal: unable to read tree (bbd3c9790981144a4e74ef5403ee71d77a852904)

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@asood-rh

asood-rh commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-metal-ds-hypershift-agent-ovn-egressip-debug

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@asood-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@asood-rh

asood-rh commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

/pj-rehearse abort

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@asood-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

The hypershift-mce-conf-os-images step requires the assisted-service-scripts
image (from: assisted-service-scripts), but it was not declared in the
release-5.0 amd64-nightly base_images, causing ErrImagePull failures
in metal-ds-hypershift-agent jobs
@openshift-ci

openshift-ci Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

The OWNERS file contains untrusted users, which makes it INVALID. The following users are mentioned in OWNERS file(s) but are untrusted for the following reasons. One way to make the user trusted is to add them as members of the openshift org. You can then trigger verification by writing /verify-owners in a comment.

  • astoycos
    • User is not a member of the org. User is not a collaborator. Satisfy at least one of these conditions to make the user trusted.
    • ci-operator/step-registry/hypershift/agent/ovn/OWNERS
    • ci-operator/step-registry/hypershift/agent/ovn/ipecho/OWNERS
    • ci-operator/step-registry/hypershift/agent/ovn/ipecho/deprovision/OWNERS
    • ci-operator/step-registry/hypershift/agent/ovn/ipecho/provision/OWNERS

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

[REHEARSALNOTIFIER]
@asood-rh: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-aws-ipi-private-shared-vpc-phz-sts-mini-perm-f28-destructive N/A periodic Ci-operator config changed
periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-azure-mag-ipi-fips-f28-destructive N/A periodic Ci-operator config changed
periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-vsphere-short-cert-rotation-f7 N/A periodic Ci-operator config changed
periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-vsphere-ipi-zones-multisubnets-external-lb-f28-destructive N/A periodic Ci-operator config changed
periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-metal-ds-ipi-ovn-f14-longrun-mco-p2 N/A periodic Ci-operator config changed
periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-ibmcloud-ipi-private-custom-cos-endpoint-fips-f28-destructive N/A periodic Ci-operator config changed
periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-nutanix-ipi-zones-external-lb-f28 N/A periodic Ci-operator config changed
periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-nutanix-ipi-zones-multi-nic-f60-destructive N/A periodic Ci-operator config changed
periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-gcp-ipi-marketplace-mini-perm-f28-destructive N/A periodic Ci-operator config changed
periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-aws-ipi-ovn-hypershift-guest-f14-destructive N/A periodic Ci-operator config changed
periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-vsphere-ipi-disk-encryption-tang-fips-f28-destructive N/A periodic Ci-operator config changed
periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-vsphere-ipi-external-lb-post-f28 N/A periodic Ci-operator config changed
periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-aws-rosa-hcp-private-proxy-int-full-f7 N/A periodic Ci-operator config changed
periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-azure-ipi-ovn-hypershift-guest-n1minor-cli-f28 N/A periodic Ci-operator config changed
periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-metal-ds-ipi-ovn-ipv4-fips-f14 N/A periodic Ci-operator config changed
periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-gcp-ipi-f7-longduration-mco-p2 N/A periodic Ci-operator config changed
periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-aws-ipi-f14-stress-olm N/A periodic Ci-operator config changed
periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-ibmcloud-ipi-proxy-private-rt-f28-destructive N/A periodic Ci-operator config changed
periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-vsphere-ipi-ovn-dualstack-f28 N/A periodic Ci-operator config changed
periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-aws-ipi-wavelength-byo-subnet-role-sts-kms-f28-destructive N/A periodic Ci-operator config changed
periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-vsphere-ipi-ovn-ipsec-mininal-permission-f28-destructive N/A periodic Ci-operator config changed
periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-metal-ds-ipi-ovn-lvms-f14 N/A periodic Ci-operator config changed
periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-azure-aks-hypershift-etcd-disk-encryption-guest-f7 N/A periodic Ci-operator config changed
periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-vsphere-upi-disc-secureboot-fips-f28-destructive N/A periodic Ci-operator config changed
periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-gcp-ipi-byo-fw-mini-perm-f28-destructive N/A periodic Ci-operator config changed

A total of 337 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs.

A full list of affected jobs can be found here

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@asood-rh

asood-rh commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-metal-ds-hypershift-agent-ovn-egressip-debug

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@asood-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@asood-rh

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-metal-ds-hypershift-agent-ovn-egressip-debug

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@asood-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@asood-rh

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-metal-ds-hypershift-agent-ovn-egressip-debug

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@asood-rh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci

openshift-ci Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

@asood-rh: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/rehearse/periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-metal-ds-hypershift-agent-ovn-egressip-debug a4e1b91 link unknown /pj-rehearse periodic-ci-openshift-openshift-tests-private-release-5.0-amd64-nightly-metal-ds-hypershift-agent-ovn-egressip-debug

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/invalid-owners-file Indicates that a PR should not merge because it has an invalid OWNERS file in it. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant